253 research outputs found
DisWOT: Student Architecture Search for Distillation WithOut Training
Knowledge distillation (KD) is an effective training strategy to improve the
lightweight student models under the guidance of cumbersome teachers. However,
the large architecture difference across the teacher-student pairs limits the
distillation gains. In contrast to previous adaptive distillation methods to
reduce the teacher-student gap, we explore a novel training-free framework to
search for the best student architectures for a given teacher. Our work first
empirically show that the optimal model under vanilla training cannot be the
winner in distillation. Secondly, we find that the similarity of feature
semantics and sample relations between random-initialized teacher-student
networks have good correlations with final distillation performances. Thus, we
efficiently measure similarity matrixs conditioned on the semantic activation
maps to select the optimal student via an evolutionary algorithm without any
training. In this way, our student architecture search for Distillation WithOut
Training (DisWOT) significantly improves the performance of the model in the
distillation stage with at least 180 training acceleration.
Additionally, we extend similarity metrics in DisWOT as new distillers and
KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201
demonstrate that our technique achieves state-of-the-art results on different
search spaces. Our project and code are available at
https://lilujunai.github.io/DisWOT-CVPR2023/.Comment: Accepted by CVPR202
NORM: Knowledge Distillation via N-to-One Representation Matching
Existing feature distillation methods commonly adopt the One-to-one
Representation Matching between any pre-selected teacher-student layer pair. In
this paper, we present N-to-One Representation (NORM), a new two-stage
knowledge distillation method, which relies on a simple Feature Transform (FT)
module consisting of two linear layers. In view of preserving the intact
information learnt by the teacher network, during training, our FT module is
merely inserted after the last convolutional layer of the student network. The
first linear layer projects the student representation to a feature space
having N times feature channels than the teacher representation from the last
convolutional layer, and the second linear layer contracts the expanded output
back to the original feature space. By sequentially splitting the expanded
student representation into N non-overlapping feature segments having the same
number of feature channels as the teacher's, they can be readily forced to
approximate the intact teacher representation simultaneously, formulating a
novel many-to-one representation matching mechanism conditioned on a single
teacher-student layer pair. After training, such an FT module will be naturally
merged into the subsequent fully connected layer thanks to its linear property,
introducing no extra parameters or architectural modifications to the student
network at inference. Extensive experiments on different visual recognition
benchmarks demonstrate the leading performance of our method. For instance, the
ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches
72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a
pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an
absolute improvement of 2.01%|4.63%|3.03% against the individually trained
counterpart. Code is available at https://github.com/OSVAI/NORMComment: The paper of NORM is published at ICLR 2023. Code and models are
available at https://github.com/OSVAI/NOR
Facile synthesis of superhydrophobic surface of ZnO nanoflakes: chemical coating and UV-induced wettability conversion
This work reports an oriented growth process of two-dimensional (2D) ZnO nanoflakes on aluminum substrate through a low temperature hydrothermal technique and proposes the preliminary growth mechanism. A bionic superhydrophobic surface with excellent corrosion protection over a wide pH range in both acidic and alkaline solutions was constructed by a chemical coating treatment with stearic acid (SA) molecules on ZnO nanoflakes. It is found that the superhydrophobic surface of ZnO nanoflake arrays shows a maximum water contact angle (CA) of 157° and a low sliding angle of 8°, and it can be reversibly switched to its initial superhydrophilic state under ultraviolet (UV) irradiation, which is due to the UV-induced decomposition of the coated SA molecules. This study is significant for simple and inexpensive building of large-scale 2D ZnO nanoflake arrays with special wettability which can extend the applications of ZnO films to many other important fields
Morphology-dependent field emission properties and wetting behavior of ZnO nanowire arrays
The fabrication of three kinds of ZnO nanowire arrays with different structural parameters over Au-coated silicon (100) by facile thermal evaporation of ZnS precursor is reported, and the growth mechanism are proposed based on structural analysis. Field emission (FE) properties and wetting behavior were revealed to be strongly morphology dependent. The nanowire arrays in small diameter and high aspect ratio exhibited the best FE performance showing a low turn-on field (4.1 V/ÎŒm) and a high field-enhancement factor (1745.8). The result also confirmed that keeping large air within the films was an effective way to obtain super water-repellent properties. This study indicates that the preparation of ZnO nanowire arrays in an optimum structural model is crucial to FE efficiency and wetting behavior
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
Recent end-to-end Automatic Speech Recognition (ASR) systems demonstrated the
ability to outperform conventional hybrid DNN/ HMM ASR. Aside from
architectural improvements in those systems, those models grew in terms of
depth, parameters and model capacity. However, these models also require more
training data to achieve comparable performance.
In this work, we combine freely available corpora for German speech
recognition, including yet unlabeled speech data, to a big dataset of over
h of speech data. For data preparation, we propose a two-stage approach
that uses an ASR model pre-trained with Connectionist Temporal Classification
(CTC) to boot-strap more training data from unsegmented or unlabeled training
data. Utterances are then extracted from label probabilities obtained from the
network trained with CTC to determine segment alignments. With this training
data, we trained a hybrid CTC/attention Transformer model that achieves
WER on the Tuda-DE test set, surpassing the previous baseline of
of conventional hybrid DNN/HMM ASR.Comment: Published at SPECOM 202
Fabrication and ultraviolet photoresponse characteristics of ordered SnOx (x â 0.87, 1.45, 2) nanopore films
Based on the porous anodic aluminum oxide templates, ordered SnOx nanopore films (approximately 150 nm thickness) with different x (x â 0.87, 1.45, 2) have been successfully fabricated by direct current magnetron sputtering and oxidizing annealing. Due to the high specific surface area, this ordered nanopore films exhibit a great improvement in recovery time compared to thin films for ultraviolet (UV) detection. Especially, the ordered SnOx nanopore films with lower x reveal higher UV light sensitivity and shorter current recovery time, which was explained by the higher concentration of the oxygen vacancies in this SnOx films. This work presents a potential candidate material for UV light detector
- âŠ